Image sensor based vignetting correction

ABSTRACT

A method and apparatus for constructing an inexpensive camera with vignetting correction in the camera electronic hardware. The hardware is programmable through the use of a register storing a constant value corresponding to the amount of vignetting of a particular lens used in the camera. The register value is provided to a correction circuit which implements an algorithm that corrects for the amount of vignetting of pixels depending upon the pixel position.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a Continuation Application of U.S. patentapplication Ser. No. 10/282,734, filed Oct. 28, 2002 and titled “ImageSensor Based Optical Defect Correction,” and is a ContinuationApplication of U.S. patent application Ser. No. 09/345,167, filed Jun.30, 1999 and titled “Image Sensor Based Vignetting Correction,” both ofwhich are incorporated by reference herein in their entirety for allpurposes.

BACKGROUND OF THE INVENTION

The present invention relates to digital cameras, and in particular tocorrecting vignetting effects.

The vignetting effect is when the brightness around the edges of ascreen is less than the brightness in the center of the screen, creatinga halo effect. Professional photographers will sometimes try tointentionally create this effect with an appropriate lens or screen, toproduce a softened image at the edges of a picture. However, when thiseffect appears unintentionally in digital photographs, it is annoying. Avariety of techniques have been developed to measure and addressvignetting effects.

U.S. Pat. No. 4,884,140 shows an analog circuit for providing vignettingcompensation for a video camera using a zoom lens which causesvignetting.

U.S. Pat. No. 5,434,904 shows measuring the vignetting effect for anx-ray examination apparatus by using a constant brightness image. Acorrection factor is then stored in the memory for each pixel.

U.S. Pat. No. 5,576,797 shows the detection of vignetting effects in acamera with a focus detecting device.

U.S. Pat. No. 5,381,174 shows the correcting of vignetting due tooperation of a zoom lens by using a field frequency sawtooth signal.

U.S. Pat. No. 4,816,663 shows the detection of vignetting between aphoto taking lens and focus detecting optical system.

Digital cameras for use with a personal computer for personalteleconferencing have become cheaper and cheaper. This puts pressure onthe camera manufacturers to use cheaper lenses, which have more of avignetting effect. In addition, the price pressure forces manufacturersto use fewer semiconductor chips, thus making the use of semiconductormemory to store vignetting corrections undesirable. Accordingly, thereis a need for a digital camera which is inexpensive to manufacture andalso corrects for vignetting of an inexpensive lens.

SUMMARY OF THE INVENTION

The present invention provides a method and apparatus for constructingan inexpensive camera with vignetting correction in the cameraelectronic hardware. The hardware is programmable through the use of aregister storing a constant value corresponding to the amount vignettingof a particular lens used in the camera. The register value is providedto a hardware correction circuit which implements an algorithm whichcorrects for the amount of vignetting of pixels depending upon the pixelposition.

The invention thus allows different lens suppliers or lens types to beused in cameras as they are manufactured, by simply programming adifferent computed value into the register. The correction is applied onthe raw sensor data after it is converted into digital form, near thesensor itself. Thus, the data is corrected prior to the optionalcollection of statistics on the sensor data and the compression of thedata for transmission to a host computer over a bus. The vignettingcorrection of the invention maintains the resolution of the image andallows real-time performance. It also makes more accurate statisticscollection, thus making the automatic gain control (AGC) in the host,and the automatic white balance (AWB) in the host more accurate.

For a further understanding of the nature and advantages of theinvention, reference should be made to the following description takenin conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a drawing of a pixel array and brightness curve illustratingthe vignetting effect.

FIG. 2 is a block diagram of a camera and computer system withvignetting correction according to one embodiment of the invention.

FIG. 3 is a block diagram of one embodiment of a hardware vignettingcorrection circuit according to the invention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

FIG. 1 illustrates a pixel array 11 and a corresponding brightness curve13 over the Y axis. A central pixel 14 with value (x₀, y₀) correspondsto the point of maximum brightness of the lens. A second example pixel15 is near the minimum brightness, showing a vignetting effect of havingthe difference between curve 13 and a maximum brightness value 16. Thepresent invention corrects this vignetting effect in both X and Ydirections.

In one embodiment, a threshold indicated by r_(T) is illustrated withinwhich the curve 13 is near the maximum value 16, and no vignettingcorrection need be applied. Alternately, the correction can be appliedto all of the pixels regardless to avoid any transition effects at thepoints r_(T).

FIG. 2 illustrates a camera and computer system implementing thevignetting correction of the invention. Shown is a video camera 17 andhost computer 18. Video camera 17 includes a lens 10 and a photo sensorarray, such as a CMOS or CCD array 12, and is connected to a USB(Universal Serial Bus) 20.

The signals from the CMOS chip are provided to an analog front end 40which includes a differential amplifier or correlated double sampler(CDS) 42, which provides the analog signals to an automatic gain control(AGC) circuit 44. The data is then provided to an analog-to-digitalconverter (ADC) 46.

The digitized signals, rather than being stored in a frame buffer memoryas in the prior art, are then provided directly to a digital processingcircuit 48. They are first provided to a video digital signal processor50 which performs the compression, cropping, scaling and other functionson the data, as well as digital filtering.

Data from video DSP 50 is provided to a bus interface 54 which includesa bus controller buffer 56 and a bus controller 58. Preferably, buscontroller buffer 56 is capable of storing two USB frames at the maximumpossible rate, or 2 Kbytes. Depending upon the frame rate, compressionrate, etc., this may typically vary up to 10 lines, or even to 15-20lines if decimation is done.

Bus controller 58 provides a control signal on a line 60 to a timinggenerator 62. Timing generator 62 provides clocking signals on line 64to CMOS chip 12. Clocking signals 64 include the horizontal and verticaltransfer pulses. In one embodiment, the analog front end 40 and thetiming generator 62 are integrated onto the same semiconductor chipsubstrate 41 as CMOS array 12.

DSP unit 50 includes a vignetting correction circuit 70 according to thepresent invention. This circuit utilizes a constant value in a constantregister 72 to perform the vignetting correction. Depending upon theparticular lens 10 which is chosen, its specified vignetting amount canbe used to provide constant 72. Thus, different lenses could be used inmanufacturing a camera by simply programming a different constant valueinto register 72. This allows multiple sources to be used for thelenses, and allows compensation for changes in lens quality from lot tolot or manufacturer to manufacturer.

As can be seen, the vignetting correction is done before a statisticsunit 74 collects statistics which are than transmitted to the hostprocessor 18. In addition, it is prior to any compression performed inbus interface 54.

On the host side of bus 20, host 18 includes a color processing unit 76which can perform AGC and AWB (automatic white balance) adjustmentsbased on the statistics provided it. Since the vignetting correction isapplied before the statistics are taken, this processing will be moreaccurate. In addition, the application of the vignetting correctionprior to compression and transmission over the bus also improves theaccuracy of the signal. After color processing, the signal is providedto a display 78 as shown.

An embodiment of an equation for implementation and hardware to performthe vignetting correction is set forth below. The vignetting defect of alow quality lens may be modeled as pass through in the middle of thelens and a parabolic curve near the edge of the lens. The transformfunction is isotropic. Let p_(i) and p_(o) be the pixel value before andafter vignetting correction, and x₀ and y₀ be the coordinate of thecenter of the lens (should be the center of the image if the lens ismounted properly). Then, $\begin{matrix}{p_{o} = {p_{i}*\left( {{a*r^{2}} + 1} \right)}} \\{= {p_{i}*\left( {{a*\left( {\left( {x - x_{0}} \right)^{2} + \left( {y - y_{0}} \right)^{2}} \right)} + 1} \right.}}\end{matrix}$

“a” is a small positive constant. To be on the safe side, a maximum of50% vignetting can be corrected, which means:1.0=0.5*(a*(176*176+144*144)+1)a=1.9338*10⁻⁵

Normalized by 2²⁶, this will give a=1298. By limiting a to 0 through1023, and performing the inverse computation, the maximum correctablevignetting is 56%.

Note that in the formula given above, the pixel aspect ratio is nottaken into account. For square pixel, the equation is exact, for 12/11pixel aspect ratio, the result is slightly off. Also, a will be scaledby ¼ if the sensor format is non-scaled QCIF (subsampled QCIF from CIFby leaving out every other pixels in both horizontal and verticaldirections).

If y_(i) is 10 bits, (x-x₀)² and (y-y₀)² is 16 bits (for CIF size), andy₀ is 10 bits, this determines that a needs to be 10 bit (0 through1023).

FIG. 3 is a block diagram of one embodiment of a hardware system forimplementing the equation described above. In particular, an optionalselection circuit 80 is applied to the select input of a multiplexer 82.Where the pixel position is less than the threshold r₂, the input pixelpi on line 84 is simply passed through to the output pixel, p₀, on line86. If it is greater than the threshold, a vignetting correction isapplied by implementing the formula above with the hardware shown inFIG. 3.

In particular, the current x value of the pixel, xi, is applied toarithmetic circuit 88, where the central pixel, x0, is subtracted fromit. This value is then squared by applying its input twice, through amultiplexer 90, to a multiply circuit 92, where it is accumulated in anadd/accumulator 94. Similarly, the y value yi is provided to a circuit96 93, where the difference from the center y value, y0, is determined.This is similarly applied as two values through mux 90 to a multiplier92, where it is squared and provided to add/accumulator 94, where it isadded to the squared x value. The use of the multiplexer simply allows,by time division multiplexing, the same multiplier 92 to be used forboth calculations. Obviously, an alternative would be to provide twoseparate multiply units.

The output of accumulator 94 is then provided to a second multiplier 96,where it is multiplied by the constant value from register 72. The valueof one is then subtracted in a unit 98, and this value is multiplied bythe pixel value, p_(i), in a multiplier 100. The output is provided on aline 102 through multiplexer 82 to output line 86. Obviously, variationsof the circuitry could be provided, such as using the same multiplycircuit for multipliers 92, 96 and 100, with the multiplier being usedfor different purposes in different time slots.

Optional circuit 80 provides a comparator 104 which compares the outputof add/accumulator 94 (the radius value of the x²+y² values) to thethreshold radius in a register 106.

Preferably, the constant a has the same number of bits as the pixelvalue, p_(i). Thus, if p_(i) is 10 bits, while the x and y values are 16bits, the constant a would be 10 bits. Also, a can be scaled by ¼ if thesensor format is non-scaled QCIF (cropped). Thus, correction is providedwhere the image is cropped before the vignetting correction.

In addition, correction can be done for the pixel aspect ratio bymultiplying the y value by 11/12 where a TV will be used.

In one embodiment, the correction will take place only outside apredefined diameter. This assumes that the lens performs satisfactorilyaround the center of the lens, which is often a reasonable assumption.By doing so, a more accurate correction model can be derived. Let r_(T)be the diameter within which the vignetting defect can be ignored. Then,If r<=r _(T),p ₀ =p _(i),if r>=r _(T),$\begin{matrix}{p_{o} = {p_{i}*\left( {{a*\left( {r^{2} - r_{T}^{2}} \right)} + 1} \right)}} \\{= {p_{i}^{*}\left( {{a*\left( {\left( {\left( {x - x_{0}} \right)^{2} + \left( {y - y_{0}} \right)^{2}} \right) - \left( {\left( {x_{T} - x_{0}} \right)^{2} + \left( {y_{T} - y_{0}} \right)^{2}} \right)} \right)} + 1} \right)}}\end{matrix}$

The present invention thus allows variation in lenses and also cheaperlenses to be used. It improves the AGC and AWB in the host computer.Additionally, it allows better object motion tracking. Object motion isusually done by assuming the luminance value is the same as the objectmoves to the edge of a sensor, and thus tracking by looking for the sameluminance value. Obviously, vignetting effects can thwart the efforts tolocate the object. By applying the vignetting correction close to thesensor, this can be overcome. The invention also provides overallimproved video quality.

Also, due to the need for image compression for transmitting over a bus,such as a universal serial bus (USB), having the host do vignettingcorrections will result in less precision. Providing vignettingcorrection on the sensor side of the bus in the present inventionprovides the correction prior to compression to transmit over the bus.

As will be understood by those of skill in the art, the presentinvention may be embodied in other specific forms without departing fromthe essential characteristics thereof. Accordingly, the foregoingdescription is intended to be illustrative, but not limiting, of thescope of the invention which is set forth in the following claims.

1. A method for manufacturing a camera which connects to a computer,comprising: determining a vignetting effect of a first lens; installingsaid first lens in a first camera housing; determining a vignettingeffect of a second lens; installing said second lens in a second camerahousing; installing in each housing a photosensor array mounted behindsaid lens; coupling an analog processing circuit to each saidphotosensor array; coupling an analog to digital converter to each saidanalog processing circuit; coupling a digital processing circuit to anoutput of each said analog to digital converter, said digital processingcircuit including a register for storing a constant corresponding to avignetting rating of said lens, a correction circuit, coupled to saidregister, for adjusting a value of selected pixels according to aformula using said constant, to correct for vignetting of said lens;coupling a bus interface circuit to an output of each said digitalprocessing circuit for connecting to a bus for transfer of data to saidcomputer; storing a first constant in said register in said first camerahousing, said first constant corresponding to said vignetting effect ofsaid first lens; storing a second constant in said register in saidsecond camera housing, said second constant corresponding to saidvignetting effect of said second lens.
 2. The method of claim 1 whereinsaid correction circuit does not adjust the value of pixels within aselected distance of a center of said lens.
 3. The method of claim 1wherein said correction circuit also corrects for a pixel aspect ratio.4. The method of claim 1 wherein said constant has a number of bitsequal to a number of bits used to represent a brightness of saidselected pixels.
 5. The method of claim 1 wherein said correctioncircuit further comprises a scaling input for scaling said constant. 6.A camera for connecting to a computer, comprising: a lens; a photosensorarray mounted behind said lens; an analog processing circuit coupled tosaid photosensor array; an analog to digital converter coupled to saidanalog processing circuit; a digital processing circuit coupled to anoutput of said analog to digital converter, said digital processingcircuit including a register for storing a constant corresponding to avignetting rating of said lens, a correction circuit, coupled to saidregister, for adjusting a value of selected pixels according to aformula using said constant, to correct for vignetting of said lens; anda bus interface circuit coupled to an output of said digital processingcircuit for connecting to a bus for transfer of data to said computer.7. The camera of claim 6 wherein said correction circuit does not adjustthe value of pixels within a selected distance of a center of said lens.8. The camera of claim 6 wherein said correction circuit also correctsfor a pixel aspect ratio.
 9. The camera of claim 6 wherein said constanthas a number of bits equal to a number of bits used to represent abrightness of said selected pixels.
 10. The camera of claim 6 furthercomprising a scaling input to said correction circuit, for scaling saidconstant.